{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment: \n",
    "\n",
    "- Opposite of Hebbian Learning: Hebbian Learning by pruning the highest coactivation, instead of the lowest. \n",
    "- Opposite of Hebbian Growth: growth connections by allowing gradient flow on connections with the lowest coactivation, instead of the highest\n",
    "\n",
    "#### Motivation.\n",
    "\n",
    "- Verify the relevance of highest coactivated units, by checking their impact on the model when they are pruned\n",
    "- Verify the relevance of lowest coactivated units, by checking their impact on the model when they are added to the model\n",
    "\n",
    "#### Conclusions:\n",
    "\n",
    "- The opposite logic of hebbian pruning, when weight pruning is set to 0, clearly affects the model performance.\n",
    "- Acc when full pruning is done at each state is 0.965 {(1,0), (0,1), (1,1)}\n",
    "- Acc with no pruning is 0.977 {(0,0)}\n",
    "- Best acc is still with only magnitude based pruning {(0,0.2), (0, 0.4)}\n",
    "- Opposite of hebbian prunning (removing connections with highest coactivation) only is harmful to the model, with acc equal or worst than full pruning, even with as low as 0.2 pruning\n",
    "- Opposite random growth (adding connections with lowest activation) reduces acc by ~ 0.02"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import os\n",
    "import glob\n",
    "import tabulate\n",
    "import pprint\n",
    "import click\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from ray.tune.commands import *\n",
    "from dynamic_sparse.common.browser import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and check data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "exps = ['neurips_debug_test10', 'neurips_debug_test11']\n",
    "paths = [os.path.expanduser(\"~/nta/results/{}\".format(e)) for e in exps]\n",
    "df = load_many(paths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Experiment Name</th>\n",
       "      <th>train_acc_max</th>\n",
       "      <th>train_acc_max_epoch</th>\n",
       "      <th>train_acc_min</th>\n",
       "      <th>train_acc_min_epoch</th>\n",
       "      <th>train_acc_median</th>\n",
       "      <th>train_acc_last</th>\n",
       "      <th>val_acc_max</th>\n",
       "      <th>val_acc_max_epoch</th>\n",
       "      <th>val_acc_min</th>\n",
       "      <th>...</th>\n",
       "      <th>momentum</th>\n",
       "      <th>network</th>\n",
       "      <th>num_classes</th>\n",
       "      <th>on_perc</th>\n",
       "      <th>optim_alg</th>\n",
       "      <th>pruning_early_stop</th>\n",
       "      <th>test_noise</th>\n",
       "      <th>use_kwinners</th>\n",
       "      <th>weight_decay</th>\n",
       "      <th>weight_prune_perc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0_hebbian_prune_perc=None,weight_prune_perc=None</td>\n",
       "      <td>0.987767</td>\n",
       "      <td>29</td>\n",
       "      <td>0.921683</td>\n",
       "      <td>0</td>\n",
       "      <td>0.984892</td>\n",
       "      <td>0.987767</td>\n",
       "      <td>0.9764</td>\n",
       "      <td>17</td>\n",
       "      <td>0.9629</td>\n",
       "      <td>...</td>\n",
       "      <td>0.9</td>\n",
       "      <td>MLPHeb</td>\n",
       "      <td>10</td>\n",
       "      <td>0.2</td>\n",
       "      <td>SGD</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1_hebbian_prune_perc=0.2,weight_prune_perc=None</td>\n",
       "      <td>0.931967</td>\n",
       "      <td>1</td>\n",
       "      <td>0.852817</td>\n",
       "      <td>22</td>\n",
       "      <td>0.876142</td>\n",
       "      <td>0.866717</td>\n",
       "      <td>0.9653</td>\n",
       "      <td>0</td>\n",
       "      <td>0.9016</td>\n",
       "      <td>...</td>\n",
       "      <td>0.9</td>\n",
       "      <td>MLPHeb</td>\n",
       "      <td>10</td>\n",
       "      <td>0.2</td>\n",
       "      <td>SGD</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2_hebbian_prune_perc=0.4,weight_prune_perc=None</td>\n",
       "      <td>0.925267</td>\n",
       "      <td>0</td>\n",
       "      <td>0.842883</td>\n",
       "      <td>13</td>\n",
       "      <td>0.868217</td>\n",
       "      <td>0.860283</td>\n",
       "      <td>0.9648</td>\n",
       "      <td>0</td>\n",
       "      <td>0.9008</td>\n",
       "      <td>...</td>\n",
       "      <td>0.9</td>\n",
       "      <td>MLPHeb</td>\n",
       "      <td>10</td>\n",
       "      <td>0.2</td>\n",
       "      <td>SGD</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3_hebbian_prune_perc=0.6,weight_prune_perc=None</td>\n",
       "      <td>0.922650</td>\n",
       "      <td>0</td>\n",
       "      <td>0.810317</td>\n",
       "      <td>22</td>\n",
       "      <td>0.869442</td>\n",
       "      <td>0.854883</td>\n",
       "      <td>0.9612</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8888</td>\n",
       "      <td>...</td>\n",
       "      <td>0.9</td>\n",
       "      <td>MLPHeb</td>\n",
       "      <td>10</td>\n",
       "      <td>0.2</td>\n",
       "      <td>SGD</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4_hebbian_prune_perc=0.8,weight_prune_perc=None</td>\n",
       "      <td>0.926633</td>\n",
       "      <td>0</td>\n",
       "      <td>0.208517</td>\n",
       "      <td>28</td>\n",
       "      <td>0.878492</td>\n",
       "      <td>0.397800</td>\n",
       "      <td>0.9647</td>\n",
       "      <td>0</td>\n",
       "      <td>0.2306</td>\n",
       "      <td>...</td>\n",
       "      <td>0.9</td>\n",
       "      <td>MLPHeb</td>\n",
       "      <td>10</td>\n",
       "      <td>0.2</td>\n",
       "      <td>SGD</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    Experiment Name  train_acc_max  \\\n",
       "0  0_hebbian_prune_perc=None,weight_prune_perc=None       0.987767   \n",
       "1   1_hebbian_prune_perc=0.2,weight_prune_perc=None       0.931967   \n",
       "2   2_hebbian_prune_perc=0.4,weight_prune_perc=None       0.925267   \n",
       "3   3_hebbian_prune_perc=0.6,weight_prune_perc=None       0.922650   \n",
       "4   4_hebbian_prune_perc=0.8,weight_prune_perc=None       0.926633   \n",
       "\n",
       "   train_acc_max_epoch  train_acc_min  train_acc_min_epoch  train_acc_median  \\\n",
       "0                   29       0.921683                    0          0.984892   \n",
       "1                    1       0.852817                   22          0.876142   \n",
       "2                    0       0.842883                   13          0.868217   \n",
       "3                    0       0.810317                   22          0.869442   \n",
       "4                    0       0.208517                   28          0.878492   \n",
       "\n",
       "   train_acc_last  val_acc_max  val_acc_max_epoch  val_acc_min  ...  momentum  \\\n",
       "0        0.987767       0.9764                 17       0.9629  ...       0.9   \n",
       "1        0.866717       0.9653                  0       0.9016  ...       0.9   \n",
       "2        0.860283       0.9648                  0       0.9008  ...       0.9   \n",
       "3        0.854883       0.9612                  0       0.8888  ...       0.9   \n",
       "4        0.397800       0.9647                  0       0.2306  ...       0.9   \n",
       "\n",
       "   network  num_classes  on_perc optim_alg  pruning_early_stop  test_noise  \\\n",
       "0   MLPHeb           10      0.2       SGD                   0       False   \n",
       "1   MLPHeb           10      0.2       SGD                   0       False   \n",
       "2   MLPHeb           10      0.2       SGD                   0       False   \n",
       "3   MLPHeb           10      0.2       SGD                   0       False   \n",
       "4   MLPHeb           10      0.2       SGD                   0       False   \n",
       "\n",
       "   use_kwinners weight_decay weight_prune_perc  \n",
       "0         False       0.0001               NaN  \n",
       "1         False       0.0001               NaN  \n",
       "2         False       0.0001               NaN  \n",
       "3         False       0.0001               NaN  \n",
       "4         False       0.0001               NaN  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace hebbian prine\n",
    "df['hebbian_prune_perc'] = df['hebbian_prune_perc'].replace(np.nan, 0.0, regex=True)\n",
    "df['weight_prune_perc'] = df['weight_prune_perc'].replace(np.nan, 0.0, regex=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Experiment Name', 'train_acc_max', 'train_acc_max_epoch',\n",
       "       'train_acc_min', 'train_acc_min_epoch', 'train_acc_median',\n",
       "       'train_acc_last', 'val_acc_max', 'val_acc_max_epoch', 'val_acc_min',\n",
       "       'val_acc_min_epoch', 'val_acc_median', 'val_acc_last', 'epochs',\n",
       "       'experiment_file_name', 'trial_time', 'mean_epoch_time', 'batch_norm',\n",
       "       'data_dir', 'dataset_name', 'debug_sparse', 'debug_weights', 'device',\n",
       "       'hebbian_grow', 'hebbian_prune_perc', 'hidden_sizes', 'input_size',\n",
       "       'learning_rate', 'lr_gamma', 'lr_milestones', 'lr_scheduler', 'model',\n",
       "       'momentum', 'network', 'num_classes', 'on_perc', 'optim_alg',\n",
       "       'pruning_early_stop', 'test_noise', 'use_kwinners', 'weight_decay',\n",
       "       'weight_prune_perc'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(217, 42)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Experiment Name           1_hebbian_prune_perc=0.2,weight_prune_perc=None\n",
       "train_acc_max                                                    0.931967\n",
       "train_acc_max_epoch                                                     1\n",
       "train_acc_min                                                    0.852817\n",
       "train_acc_min_epoch                                                    22\n",
       "train_acc_median                                                 0.876142\n",
       "train_acc_last                                                   0.866717\n",
       "val_acc_max                                                        0.9653\n",
       "val_acc_max_epoch                                                       0\n",
       "val_acc_min                                                        0.9016\n",
       "val_acc_min_epoch                                                      22\n",
       "val_acc_median                                                    0.92105\n",
       "val_acc_last                                                       0.9216\n",
       "epochs                                                                 30\n",
       "experiment_file_name    /Users/lsouza/nta/results/neurips_debug_test10...\n",
       "trial_time                                                        19.7118\n",
       "mean_epoch_time                                                  0.657061\n",
       "batch_norm                                                           True\n",
       "data_dir                                        /home/ubuntu/nta/datasets\n",
       "dataset_name                                                        MNIST\n",
       "debug_sparse                                                         True\n",
       "debug_weights                                                        True\n",
       "device                                                               cuda\n",
       "hebbian_grow                                                        False\n",
       "hebbian_prune_perc                                                    0.2\n",
       "hidden_sizes                                                          100\n",
       "input_size                                                            784\n",
       "learning_rate                                                         0.1\n",
       "lr_gamma                                                              0.1\n",
       "lr_milestones                                                          60\n",
       "lr_scheduler                                                  MultiStepLR\n",
       "model                                                        DSNNMixedHeb\n",
       "momentum                                                              0.9\n",
       "network                                                            MLPHeb\n",
       "num_classes                                                            10\n",
       "on_perc                                                               0.2\n",
       "optim_alg                                                             SGD\n",
       "pruning_early_stop                                                      0\n",
       "test_noise                                                          False\n",
       "use_kwinners                                                        False\n",
       "weight_decay                                                       0.0001\n",
       "weight_prune_perc                                                       0\n",
       "Name: 1, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "model\n",
       "DSNNMixedHeb    217\n",
       "Name: model, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('model')['model'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experiment Details"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "base_exp_config = dict(\n",
    "    device=\"cuda\",\n",
    "    # dataset related\n",
    "    dataset_name=\"MNIST\",\n",
    "    data_dir=os.path.expanduser(\"~/nta/datasets\"),\n",
    "    input_size=784,\n",
    "    num_classes=10,\n",
    "    # network related\n",
    "    network=\"MLPHeb\",\n",
    "    hidden_sizes=[100, 100, 100],\n",
    "    batch_norm=True,\n",
    "    use_kwinners=tune.grid_search([True, False]),\n",
    "    # model related\n",
    "    model=\"DSNNMixedHeb\",\n",
    "    on_perc=0.2,\n",
    "    optim_alg=\"SGD\",\n",
    "    momentum=0.9,\n",
    "    weight_decay=1e-4,    \n",
    "    learning_rate=0.1,\n",
    "    lr_scheduler=\"MultiStepLR\",\n",
    "    lr_milestones=[30,60,90],\n",
    "    lr_gamma=0.1,\n",
    "    # sparse related\n",
    "    hebbian_prune_perc=tune.grid_search([0, 0.1, 0.2, 0.3, 0.4, 0.5]),\n",
    "    pruning_early_stop=0,\n",
    "    hebbian_grow=tune.grid_search([True, False]),\n",
    "    # additional validation\n",
    "    test_noise=False,\n",
    "    # debugging\n",
    "    debug_weights=True,\n",
    "    debug_sparse=True,\n",
    "    stop={\"training_iteration\": 30},\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Did any  trials failed?\n",
    "df[df[\"epochs\"]<30][\"epochs\"].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(216, 42)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Removing failed or incomplete trials\n",
    "df_origin = df.copy()\n",
    "df = df_origin[df_origin[\"epochs\"]>=30]\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "108    1\n",
       "Name: epochs, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# which ones failed?\n",
    "# failed, or still ongoing?\n",
    "df_origin['failed'] = df_origin[\"epochs\"]<30\n",
    "df_origin[df_origin['failed']]['epochs']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper functions\n",
    "def mean_and_std(s):\n",
    "    return \"{:.3f} ± {:.3f}\".format(s.mean(), s.std())\n",
    "\n",
    "def round_mean(s):\n",
    "    return \"{:.0f}\".format(round(s.mean()))\n",
    "\n",
    "stats = ['min', 'max', 'mean', 'std']\n",
    "\n",
    "def agg(columns, filter=None, round=3):\n",
    "    if filter is None:\n",
    "        return (df.groupby(columns)\n",
    "             .agg({'val_acc_max_epoch': round_mean,\n",
    "                   'val_acc_max': stats,                \n",
    "                   'model': ['count']})).round(round)\n",
    "    else:\n",
    "        return (df[filter].groupby(columns)\n",
    "             .agg({'val_acc_max_epoch': round_mean,\n",
    "                   'val_acc_max': stats,                \n",
    "                   'model': ['count']})).round(round)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### What is the impact of removing connections with highest coactivation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_grow = (df['hebbian_grow'] == False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>val_acc_max_epoch</th>\n",
       "      <th colspan=\"4\" halign=\"left\">val_acc_max</th>\n",
       "      <th>model</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>round_mean</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hebbian_prune_perc</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>21</td>\n",
       "      <td>0.964</td>\n",
       "      <td>0.982</td>\n",
       "      <td>0.977</td>\n",
       "      <td>0.006</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>12</td>\n",
       "      <td>0.961</td>\n",
       "      <td>0.980</td>\n",
       "      <td>0.973</td>\n",
       "      <td>0.007</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>15</td>\n",
       "      <td>0.959</td>\n",
       "      <td>0.980</td>\n",
       "      <td>0.973</td>\n",
       "      <td>0.008</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>17</td>\n",
       "      <td>0.957</td>\n",
       "      <td>0.980</td>\n",
       "      <td>0.973</td>\n",
       "      <td>0.009</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>16</td>\n",
       "      <td>0.960</td>\n",
       "      <td>0.981</td>\n",
       "      <td>0.974</td>\n",
       "      <td>0.008</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>15</td>\n",
       "      <td>0.964</td>\n",
       "      <td>0.982</td>\n",
       "      <td>0.975</td>\n",
       "      <td>0.008</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   val_acc_max_epoch val_acc_max                      model\n",
       "                          round_mean         min    max   mean    std count\n",
       "hebbian_prune_perc                                                         \n",
       "0.0                               21       0.964  0.982  0.977  0.006    18\n",
       "0.2                               12       0.961  0.980  0.973  0.007    18\n",
       "0.4                               15       0.959  0.980  0.973  0.008    18\n",
       "0.6                               17       0.957  0.980  0.973  0.009    18\n",
       "0.8                               16       0.960  0.981  0.974  0.008    18\n",
       "1.0                               15       0.964  0.982  0.975  0.008    18"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agg(['hebbian_prune_perc'], random_grow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>val_acc_max_epoch</th>\n",
       "      <th colspan=\"4\" halign=\"left\">val_acc_max</th>\n",
       "      <th>model</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>round_mean</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight_prune_perc</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>4</td>\n",
       "      <td>0.957</td>\n",
       "      <td>0.978</td>\n",
       "      <td>0.966</td>\n",
       "      <td>0.006</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>22</td>\n",
       "      <td>0.976</td>\n",
       "      <td>0.982</td>\n",
       "      <td>0.979</td>\n",
       "      <td>0.002</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>21</td>\n",
       "      <td>0.977</td>\n",
       "      <td>0.982</td>\n",
       "      <td>0.979</td>\n",
       "      <td>0.001</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>23</td>\n",
       "      <td>0.978</td>\n",
       "      <td>0.981</td>\n",
       "      <td>0.979</td>\n",
       "      <td>0.001</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>23</td>\n",
       "      <td>0.976</td>\n",
       "      <td>0.981</td>\n",
       "      <td>0.979</td>\n",
       "      <td>0.001</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>3</td>\n",
       "      <td>0.959</td>\n",
       "      <td>0.967</td>\n",
       "      <td>0.963</td>\n",
       "      <td>0.002</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  val_acc_max_epoch val_acc_max                      model\n",
       "                         round_mean         min    max   mean    std count\n",
       "weight_prune_perc                                                         \n",
       "0.0                               4       0.957  0.978  0.966  0.006    18\n",
       "0.2                              22       0.976  0.982  0.979  0.002    18\n",
       "0.4                              21       0.977  0.982  0.979  0.001    18\n",
       "0.6                              23       0.978  0.981  0.979  0.001    18\n",
       "0.8                              23       0.976  0.981  0.979  0.001    18\n",
       "1.0                               3       0.959  0.967  0.963  0.002    18"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agg(['weight_prune_perc'], random_grow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### What is the optimal combination of both"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>weight_prune_perc</th>\n",
       "      <th>0.0</th>\n",
       "      <th>0.2</th>\n",
       "      <th>0.4</th>\n",
       "      <th>0.6</th>\n",
       "      <th>0.8</th>\n",
       "      <th>1.0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hebbian_prune_perc</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.981 ± 0.001</td>\n",
       "      <td>0.981 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.980 ± 0.000</td>\n",
       "      <td>0.965 ± 0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>0.964 ± 0.001</td>\n",
       "      <td>0.978 ± 0.002</td>\n",
       "      <td>0.978 ± 0.000</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.963 ± 0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>0.963 ± 0.001</td>\n",
       "      <td>0.978 ± 0.001</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.962 ± 0.003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>0.960 ± 0.003</td>\n",
       "      <td>0.979 ± 0.000</td>\n",
       "      <td>0.979 ± 0.002</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.963 ± 0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>0.965 ± 0.000</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.979 ± 0.000</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.961 ± 0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.964 ± 0.000</td>\n",
       "      <td>0.981 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.965 ± 0.002</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "weight_prune_perc             0.0            0.2            0.4  \\\n",
       "hebbian_prune_perc                                                \n",
       "0.0                 0.977 ± 0.001  0.981 ± 0.001  0.981 ± 0.001   \n",
       "0.2                 0.964 ± 0.001  0.978 ± 0.002  0.978 ± 0.000   \n",
       "0.4                 0.963 ± 0.001  0.978 ± 0.001  0.979 ± 0.001   \n",
       "0.6                 0.960 ± 0.003  0.979 ± 0.000  0.979 ± 0.002   \n",
       "0.8                 0.965 ± 0.000  0.980 ± 0.001  0.980 ± 0.001   \n",
       "1.0                 0.964 ± 0.000  0.981 ± 0.001  0.980 ± 0.001   \n",
       "\n",
       "weight_prune_perc             0.6            0.8            1.0  \n",
       "hebbian_prune_perc                                               \n",
       "0.0                 0.980 ± 0.001  0.980 ± 0.000  0.965 ± 0.001  \n",
       "0.2                 0.979 ± 0.001  0.977 ± 0.001  0.963 ± 0.001  \n",
       "0.4                 0.979 ± 0.001  0.979 ± 0.001  0.962 ± 0.003  \n",
       "0.6                 0.980 ± 0.001  0.979 ± 0.001  0.963 ± 0.002  \n",
       "0.8                 0.979 ± 0.000  0.979 ± 0.001  0.961 ± 0.001  \n",
       "1.0                 0.980 ± 0.001  0.980 ± 0.001  0.965 ± 0.002  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.pivot_table(df[random_grow], \n",
    "              index='hebbian_prune_perc',\n",
    "              columns='weight_prune_perc',\n",
    "              values='val_acc_max',\n",
    "              aggfunc=mean_and_std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The opposite logic of hebbian pruning, when weight pruning is set to 0, clearly affects the model performance.\n",
    "- Acc when full pruning is done at each state is 0.965 {(1,0), (0,1), (1,1)}\n",
    "- Acc with no pruning is 0.977 {(0,0)}\n",
    "- Best acc is still with only magnitude based pruning {(0,0.2), (0, 0.4)}\n",
    "- Opposite of hebbian prunning (removing connections with highest coactivation) only is harmful to the model, with acc equal or worst than full pruning, even with as low as 0.2 pruning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### What is the impact of the adding connections with lowest coactivation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>val_acc_max_epoch</th>\n",
       "      <th colspan=\"4\" halign=\"left\">val_acc_max</th>\n",
       "      <th>model</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>round_mean</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hebbian_grow</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>False</th>\n",
       "      <td>16</td>\n",
       "      <td>0.957</td>\n",
       "      <td>0.982</td>\n",
       "      <td>0.974</td>\n",
       "      <td>0.008</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>True</th>\n",
       "      <td>13</td>\n",
       "      <td>0.956</td>\n",
       "      <td>0.979</td>\n",
       "      <td>0.972</td>\n",
       "      <td>0.007</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             val_acc_max_epoch val_acc_max                      model\n",
       "                    round_mean         min    max   mean    std count\n",
       "hebbian_grow                                                         \n",
       "False                       16       0.957  0.982  0.974  0.008   108\n",
       "True                        13       0.956  0.979  0.972  0.007   108"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# with and without hebbian grow\n",
    "agg('hebbian_grow')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>weight_prune_perc</th>\n",
       "      <th>0.0</th>\n",
       "      <th>0.2</th>\n",
       "      <th>0.4</th>\n",
       "      <th>0.6</th>\n",
       "      <th>0.8</th>\n",
       "      <th>1.0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hebbian_grow</th>\n",
       "      <th>hebbian_prune_perc</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"6\" valign=\"top\">False</th>\n",
       "      <th>0.0</th>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.981 ± 0.001</td>\n",
       "      <td>0.981 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.980 ± 0.000</td>\n",
       "      <td>0.965 ± 0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>0.964 ± 0.001</td>\n",
       "      <td>0.978 ± 0.002</td>\n",
       "      <td>0.978 ± 0.000</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.963 ± 0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>0.963 ± 0.001</td>\n",
       "      <td>0.978 ± 0.001</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.962 ± 0.003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>0.960 ± 0.003</td>\n",
       "      <td>0.979 ± 0.000</td>\n",
       "      <td>0.979 ± 0.002</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.963 ± 0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>0.965 ± 0.000</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.979 ± 0.000</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.961 ± 0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.964 ± 0.000</td>\n",
       "      <td>0.981 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.980 ± 0.001</td>\n",
       "      <td>0.965 ± 0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"6\" valign=\"top\">True</th>\n",
       "      <th>0.0</th>\n",
       "      <td>0.976 ± 0.001</td>\n",
       "      <td>0.978 ± 0.001</td>\n",
       "      <td>0.979 ± 0.001</td>\n",
       "      <td>0.975 ± 0.001</td>\n",
       "      <td>0.974 ± 0.001</td>\n",
       "      <td>0.963 ± 0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2</th>\n",
       "      <td>0.961 ± 0.001</td>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.978 ± 0.001</td>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.975 ± 0.000</td>\n",
       "      <td>0.964 ± 0.003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.4</th>\n",
       "      <td>0.962 ± 0.002</td>\n",
       "      <td>0.978 ± 0.001</td>\n",
       "      <td>0.976 ± 0.000</td>\n",
       "      <td>0.976 ± 0.000</td>\n",
       "      <td>0.974 ± 0.001</td>\n",
       "      <td>0.962 ± 0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6</th>\n",
       "      <td>0.963 ± 0.001</td>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.977 ± 0.000</td>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.974 ± 0.001</td>\n",
       "      <td>0.961 ± 0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8</th>\n",
       "      <td>0.963 ± 0.003</td>\n",
       "      <td>0.977 ± 0.000</td>\n",
       "      <td>0.976 ± 0.001</td>\n",
       "      <td>0.976 ± 0.001</td>\n",
       "      <td>0.974 ± 0.001</td>\n",
       "      <td>0.961 ± 0.004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.958 ± 0.002</td>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.977 ± 0.001</td>\n",
       "      <td>0.976 ± 0.001</td>\n",
       "      <td>0.972 ± 0.001</td>\n",
       "      <td>0.961 ± 0.002</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "weight_prune_perc                          0.0            0.2            0.4  \\\n",
       "hebbian_grow hebbian_prune_perc                                                \n",
       "False        0.0                 0.977 ± 0.001  0.981 ± 0.001  0.981 ± 0.001   \n",
       "             0.2                 0.964 ± 0.001  0.978 ± 0.002  0.978 ± 0.000   \n",
       "             0.4                 0.963 ± 0.001  0.978 ± 0.001  0.979 ± 0.001   \n",
       "             0.6                 0.960 ± 0.003  0.979 ± 0.000  0.979 ± 0.002   \n",
       "             0.8                 0.965 ± 0.000  0.980 ± 0.001  0.980 ± 0.001   \n",
       "             1.0                 0.964 ± 0.000  0.981 ± 0.001  0.980 ± 0.001   \n",
       "True         0.0                 0.976 ± 0.001  0.978 ± 0.001  0.979 ± 0.001   \n",
       "             0.2                 0.961 ± 0.001  0.977 ± 0.001  0.978 ± 0.001   \n",
       "             0.4                 0.962 ± 0.002  0.978 ± 0.001  0.976 ± 0.000   \n",
       "             0.6                 0.963 ± 0.001  0.977 ± 0.001  0.977 ± 0.000   \n",
       "             0.8                 0.963 ± 0.003  0.977 ± 0.000  0.976 ± 0.001   \n",
       "             1.0                 0.958 ± 0.002  0.977 ± 0.001  0.977 ± 0.001   \n",
       "\n",
       "weight_prune_perc                          0.6            0.8            1.0  \n",
       "hebbian_grow hebbian_prune_perc                                               \n",
       "False        0.0                 0.980 ± 0.001  0.980 ± 0.000  0.965 ± 0.001  \n",
       "             0.2                 0.979 ± 0.001  0.977 ± 0.001  0.963 ± 0.001  \n",
       "             0.4                 0.979 ± 0.001  0.979 ± 0.001  0.962 ± 0.003  \n",
       "             0.6                 0.980 ± 0.001  0.979 ± 0.001  0.963 ± 0.002  \n",
       "             0.8                 0.979 ± 0.000  0.979 ± 0.001  0.961 ± 0.001  \n",
       "             1.0                 0.980 ± 0.001  0.980 ± 0.001  0.965 ± 0.002  \n",
       "True         0.0                 0.975 ± 0.001  0.974 ± 0.001  0.963 ± 0.002  \n",
       "             0.2                 0.977 ± 0.001  0.975 ± 0.000  0.964 ± 0.003  \n",
       "             0.4                 0.976 ± 0.000  0.974 ± 0.001  0.962 ± 0.001  \n",
       "             0.6                 0.977 ± 0.001  0.974 ± 0.001  0.961 ± 0.002  \n",
       "             0.8                 0.976 ± 0.001  0.974 ± 0.001  0.961 ± 0.004  \n",
       "             1.0                 0.976 ± 0.001  0.972 ± 0.001  0.961 ± 0.002  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# with and without hebbian grow\n",
    "pd.pivot_table(df, \n",
    "              index=['hebbian_grow', 'hebbian_prune_perc'],\n",
    "              columns='weight_prune_perc',\n",
    "              values='val_acc_max',\n",
    "              aggfunc=mean_and_std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Opposite random growth (adding connections with lowest activation) reduces acc by ~ 0.02\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
